Data Led Growth project | Freshworks

​

Picking the Right Experiment

To improve the efficiency of marketing spend, the performance marketing team at Freshworks employs the following tactics:

1. Audience Segmentation
2. Data-Driven Optimizations
3. A/B Testing
4. Adaptive Budget Allocations
5. Conversion Rate Optimization

​

Points 1-4 form the foundation of our performance marketing efforts. Conversion Rate Optimization (CRO) is often something that gets overlooked but if proven successful can amplify results within existing budgets by generating incremental volumes

Overview

• Performance marketing teams at Freshworks are split as follows:
• • Paid Marketing - Generating paid traffic via Google, Bing, Paid Social and Marketplaces
  • Organic Marketing - Generating organic traffic via ranking content on search
  • Affiliate Marketing - Generating affiliate traffic by partnering with affiliate partners
  • Referral - Generating traffic from internal (website) & external referral sources
• While these teams own individual OKRs based on their surface area, there is a channel that is often ignored as uncontrollable - DIRECT
• • Direct traffic according to SEMRUSH is the traffic that lands by typing your URL directly into their browser
  • Users coming from direct traffic can be assumed as people who are either brand-aware aware / product-aware

​

Hypothesis

​

Goal

• Our goal from this experiment is to improve the yield from returning DIRECT users

​

​

Success Metrics

• Success in this experiment is defined by
• • Primary Success Metric - Lift in Web Conversion Rate ​
  • • Best Case Scenario - Increase in WCR by 10%
    • Worst Case Scenario - Increase in WCR by 1-2% (flat)
    • • In this scenario, given the WCR did not decrease we would like to re-run the experiment by optimizing H1 & Imagery
  • Secondary Success Metrics - Lift in Visits to Signup / Demo Page
  • Health Metrics - Exit / Bounce Rates

​

Experiment Design

What are we testing?

• We are testing the first fold of the Freshdesk homepage which remains static for every channel/audience that lands on this page.
• The first fold can be capitalized by personalizing the H1 text and imagery on the right to improve web conversion rates
• Funnel Data - From GA4

​

​

There is an opportunity to improve the conversion rate of returning users by personalizing the H1 and offering a product tour

​

​

​

​

Variation Design

Following changes will be made to the variant

1. Variant H1 text and subtext to be personalized for returning users
2. The video on the right will be replaced with a product tour
3. 1. A product tour is an interactive demo of the product that explains the capabilities in 10-12 steps
4. Other elements besides the first fold (H1 & subtext and product tour) will remain the same as the control

​

Control

​

​

​

Variant ​

​

​

Audience & Sample Size

• Audience
• • Our audience for this test is all returning users from the direct channel landing on the Freshdesk homepage
• Sample Size
• • Using the Optimizely A/B test sample size calculator and adding inputs on the current baseline conversion rate, MDE, and statistical significance (95%) we get a sample size of 86,000 users
• Duration of Test
• • June 1 - July 30 or till we are statistically significant (95%), whichever is earlier

​

​

Implementation & A/A Test

• Platform - Optimizely Web Experimentation
• A/A test -
• • Connect GA4 to Optimizely to measure the success metrics mentioned above (Freshdesk signup conversion rate, visits to the signup page, bounce/exit rates)
  • Run the A/A test by keeping the control and variant the same and measuring the metrics defined above
  • Results generated were the same across the control and variant indicating experiment setup is valid

Did not have access to test this on Optimizely, hence explanatory data is added below ​

Post Experiment

SAMPLE DATA ADDED BELOW - Did not have access to test this, hence explanatory data is added below

Experiment Result

• After running the experiment until statistical significance we notice that the conversion rate of variant:
• • Signup to Freshdesk increased by +12.17% (95% SS)
  • Visiting Signup Page increased by +6.6% (95% SS)
  • Exit rate remained stable across control and variant
• Based on the results, the experiment can be stopped and declared as SUCCESS

​

​

Release Decision

SCALE the experiment since it was successful with statistical significance

Learnings

Leveraging user familiarity significantly boosts engagement. Prioritizing personalized experiences and understanding user behavior is crucial for driving conversion improvement.

Next Steps

As the experiment was successful and we decided to scale this experiment, we need the following next steps to push this experiment into production:

• Since the copy & designs are ready we need a set this as a personalization experiment for 100% of returning direct users on Optimizely - ETA is 3-4 days
• To do so, we raise a ticket with the web team and use the Optimizely tool's results page to monitor results on 100% audience

Stakeholder Management

To get buy-in for this experiment we would:

​